Aircrew Automation System and Method

ABSTRACT

An aircrew automation system and method for use in an aircraft. The aircrew automation system comprises one or more processors, an optical perception system, an actuation system, and a human-machine interface. The optical perception system monitors, in real-time, one or more cockpit instruments of the aircraft visually to generate flight situation data. The actuation system mechanically engages at least one flight control of the aircraft in response to the one or more flight commands. The human-machine interface provides an interface between a human pilot and the aircrew automation system. The human-machine interface comprises a display device to display a status of the aircraft and the actuation system.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S. patentapplication Ser. No. 15/464,786, filed on Mar. 21, 2017, entitled“Aircrew Automation System and Method,” which claims priority to U.S.Provisional Patent Application Nos.: 62/311,659, filed on Mar. 22, 2016,and 62/383,032, filed on Sep. 2, 2016, each entitled “Aircrew AutomationSystem and Method.” Each application is hereby incorporated by referencein its entirety.

STATEMENT OF GOVERNMENT INTEREST

This invention was made with government support under Contract Number:HR0011-15-0027 awarded by the Defense Advanced Research Projects Agency(DARPA). The government has certain rights in the invention.

TECHNICAL FIELD

The present disclosure relates to the field of flight control systems,methods, and apparatuses; even more particularly, to a system, method,and apparatus for providing aircraft state monitoring and/or anautomated aircrew.

BACKGROUND

Recent experience with automation in cockpits has shown that priorapproaches of adding additional functionality to flight decks increasescomplexity, causes overreliance on automation, and may not necessarilyreduce workload, especially during critical situations. An additionalchallenge is that avionics manufacturers have instituted strictrequirements-based design and change orders for any desiredimprovements, in order to provide high reliability and verifiability.Thus, conversion of legacy aircraft is generally cost prohibitive andrequires a large capital investment in requirements, verification, andtesting.

Aurora Flight Sciences Corporation of Manassas, Va. has previouslydeveloped a right-seat pilot assistant capable of operating a DiamondDA42 Twin Star during takeoff, cruise, and landing. The right-seat pilotassistant, called Centaur, can be installed into, and removed from, theDA42 without affecting the original type certificate, thus maintainingthe aircraft's original certification. Centaur includes mechanicalactuation of the primary flight controls and its own avionics suite, andmay be used with a pilot in a supervisory role or as a fully unmannedaircraft. For example, Centaur may be flown by an operator in the backseat of the airplane, directing the flight plan on a laptop. WhileCentaur offers many features, it suffers from certain drawbacks. Inparticular, (1) the Centaur hardware is not portable to other aircraft,nor is the software plug-and-pay extensible to other capabilities; (2)parts of the Centaur system are invasive and require cutting intoexisting avionics wiring in a manner very specific to the aircraft(i.e., the DA42); (3) Centaur does not allow the onboard pilot to be theoperator and to perform tasks such as directing the flight plan; and (4)Centaur does not acquire knowledge about the aircraft it is operating.

Thus, a need exists for an open architecture system that enables quickintroduction of new capabilities, increases safety, grows functionality,and reduces pilot workload—without large expense or recertification.There is also a need to provide a pilot with continuous aircraft statemonitoring and information augmentation, which can effectively serve asa digital flight engineer. An aircrew automation system, such as isdisclosed herein, addresses these needs and enables new capabilities tobe rapidly introduced with minimal certification burden while beingportable across airframes (e.g., via temporary installations). As willbe discussed, the aircrew automation system can provide significantbenefit to a variety of end-users. An example application includes theoperation of aircraft where fatigue and boredom can cause a reduction increw attentiveness, in which case the aircrew automation system reducesrisk in a flight operation by alerting the pilot and, in certaininstances, assuming control of the aircraft. Other example applicationsexist where the potential for human error currently limits extensive useof aircraft (e.g., low-altitude operations), synchronized operations,unmanned flights, unmanned formations with manned flight lead, andimproved debrief capabilities due to comprehensive data logging.

SUMMARY

The present disclosure is directed to flight control systems, methods,and apparatuses; even more particularly, to a system, method, andapparatus for providing aircraft state monitoring and/or automatedaircrew.

According to a first aspect, an aircrew automation system for use in anaircraft comprises: a core platform to operatively connect a pluralityof systems or subsystems via one or more interfaces; a human-machineinterface operatively coupled with the core platform to provide aninterface between a pilot and the aircrew automation system; a knowledgeacquisition system operatively coupled with the core platform todetermine information specific to the aircraft; and a perception systemoperatively coupled with the core platform to monitor one or morecockpit instruments of the aircraft to generate flight situation data.

According to a second aspect, an aircrew automation system for use in anaircraft comprises: a core platform to operatively connect a pluralityof systems or subsystems via one or more interfaces; a human-machineinterface operatively coupled with the core platform to provide aninterface between a pilot and the aircrew automation system; a knowledgeacquisition system operatively coupled with the core platform todetermine information specific to the aircraft; an aircraft statemonitoring system coupled with the core platform to collect or generateflight situation data; and an actuation system operatively coupled withthe core platform to actuate one or flight controls of the aircraftbased in response to commands from the core platform.

According to a third aspect, an aircrew automation system for use in anaircraft comprises: a core platform to operatively connect a pluralityof systems or subsystems via one or more interfaces; a human-machineinterface operatively coupled with the core platform to provide aninterface between a pilot and the aircrew automation system, thehuman-machine interface is configured to display a plurality of tasksvia a touch-screen display in a form of a task list; a knowledgeacquisition system operatively coupled with the core platform todetermine information specific to the aircraft; and an aircraft statemonitoring system coupled with the core platform to collect or generateflight situation data.

According to a fourth aspect, an aircrew automation system for use in anaircraft comprises: a knowledge acquisition system to determineinformation specific to the aircraft; a perception system to monitor oneor more cockpit instruments of the aircraft to generate flight situationdata; a core platform operatively connected with the knowledgeacquisition system and the perception system to determine a state of theaircraft based at least in part on said flight situation data; and ahuman-machine interface operatively coupled with the core platform todisplay the state of the aircraft.

According to a fifth aspect, an aircrew automation system for use in anaircraft comprises: a knowledge acquisition system to determineinformation specific to the aircraft; an aircraft state monitoringsystem to monitor one or more cockpit instruments of the aircraft togenerate flight situation data; a core platform operatively connectedwith the knowledge acquisition system and the aircraft state monitoringsystem to determine a state of the aircraft based at least in part onsaid flight situation data; and a human-machine interface operativelycoupled with the core platform to display the state of the aircraft.

In certain aspects, the aircrew automation system is vehicle-agnostic.

In certain aspects, the aircraft state monitoring system is a perceptionsystem.

In certain aspects, the aircraft state monitoring system is hardwired toone or more cockpit instruments.

In certain aspects, the aircrew automation system employs an operatingsystem that operates as the middleware to interconnect one or moreoperational applications or hardware interfaces.

In certain aspects, the aircrew automation system is configured toprovide a pilot with continuous aircraft state monitoring andinformation augmentation with or without taking control of the aircraft.

In certain aspects, the autopilot manager is configured to control anaircrew automation actuation module or an aircraft actuation module.

In certain aspects, the commands from the core platform are generatedbased at least in part on the state of the aircraft.

In certain aspects, the core platform displays the current aircraftstate via a display of the human-machine interface.

In certain aspects, the core platform employs an open architecture.

In certain aspects, the core platform includes a flight controllersystem and/or a mission control system.

In certain aspects, the core platform is configured to communicate oneor more commands to a flight control system of the aircraft based atleast in part on the flight situation data.

In certain aspects, the core platform is configured to develop, usinginformation collected from one or more systems coupled thereto, anunderstanding of the aircraft's systems, configurations, and proceduresnecessary to maintain safe operation.

In certain aspects, the core platform is configured to receive andaggregate the flight situation data to determine a current aircraftstate.

In certain aspects, the core platform is configured with plug and playsoftware to enable (1) removal or disability of systems or subsystems or(2) installation or activation of systems or subsystems.

In certain aspects, the flight controller system includes one or more ofan autopilot manager, a vehicle manager, a state estimation module, aperception module, or a navigation module.

In certain aspects, the flight situation data includes airspeed,altitude, and vertical speed of the aircraft.

In certain aspects, the hardware interfaces include at least one of aprimary actuation interface, a secondary actuation interface, anaircraft state interface, or an HMI interface.

In certain aspects, the human-machine interface is a tablet computer.

In certain aspects, the human-machine interface enables a pilot tocontrol and communicate with the aircrew automation system.

In certain aspects, the human-machine interface includes a tool bar witha plurality of selectable tabs.

In certain aspects, the human-machine interface includes a touch-screendisplay.

In certain aspects, the touch-screen display is compatible withnight-vision goggles.

In certain aspects, the human-machine interface is configured to displaya plurality of tasks on the touch-screen display in a form of a tasklist. In certain aspects, each of the plurality of tasks is marked aseither completed or not completed, for example, based at least in parton a pilot input via a touch-screen display.

In certain aspects, the human-machine interface is configured to displayat least one of checklists, health alerts, predictions of aircraftstate, and failure prognosis.

In certain aspects, the human-machine interface employsspeech-recognition techniques to recognize a verbal communication fromthe pilot.

In certain aspects, the human-machine interface is configured to generalverbal response to the verbal communication from the pilot.

In certain aspects, the information specific to the aircraft includeswhether the aircraft is susceptible to a particular operational error.

In certain aspects, the knowledge acquisition system is configured todetermine information specific to the aircraft to yield avehicle-agnostic aircrew automation system.

In certain aspects, the knowledge acquisition system is configured todetermine the layout of the cockpit instruments.

In certain aspects, the operational applications include at least one ofa normal flight operation application, an anomaly detection application,a contingency operation application, an intelligence, surveillance, andreconnaissance (“ISR”) application, a trend recognition application, oran aerial refueling application.

In certain aspects, the perception system includes a vision systemhaving a camera to visually monitor one or more cockpit instruments ofthe aircraft.

In certain aspects, the plurality of selectable tabs includes one ormore of a route tab, a procedures tab, a calibration tab, and anapplications tab.

In certain aspects, the aircrew automation system further comprises anactuation system operatively coupled with the core platform.

In certain aspects, the actuation system comprises a primary actuationsystem and a secondary actuation system that is separate from theprimary actuation system.

In certain aspects, the aircrew automation system is configured tocontrol of the aircraft via an/the actuation system.

In certain aspects, the primary actuation system controls actuation ofthe aircraft's primary flight controls and/or the secondary actuationsystem controls actuation of the aircraft's secondary flight controls.

In certain aspects, the aircraft's primary flight controls include oneor more stick controls or yoke controls.

In certain aspects, the aircraft's secondary flight controls include oneor more switches and/or knobs.

In certain aspects, the primary actuation system includes a frame, anarticulating arm coupled at its proximal end to the frame, and a gripperpositioned at a distal end of the articulating arm to engage at leastone of the aircraft's primary flight controls.

In certain aspects, the primary actuation system includes anarticulating arm with a gripper to engage at least one of the aircraft'sprimary flight controls.

In certain aspects, the gripper is coupled to the articulating arm via amultiple-DOF (degree of freedom) connection.

In certain aspects, the articulating arm is rotatable and slideablycoupled at its proximal end to the frame via a movable base.

In certain aspects, the secondary actuation system includes anXY-plotter defining a Y-axis and an X-axis, a tool to engage at leastone of the aircraft's secondary flight controls, and a control system tomove the tool along the Y-axis and the X-axis.

In certain aspects, the tool is a multi-tool configured with a switchactuator and a knob actuator, to engage a switch or a knob,respectively.

DRAWINGS

These and other advantages of the present disclosure may be readilyunderstood with the reference to the following specifications andattached drawings wherein:

FIG. 1a illustrates a block diagram of an example aircrew automationsystem.

FIG. 1b illustrates an example flow of information data between thesubsystems of FIG. 1 a.

FIG. 1c illustrates a block diagram of an example core platform.

FIG. 2 illustrates a diagram of an example core platform architecture.

FIG. 3a illustrates a first example human-machine interface illustratinga route application.

FIG. 3b illustrates a second example human-machine interfaceillustrating a procedural checklist and aircraft health alert screen.

FIG. 4 illustrates a block diagram of an example perception system.

FIGS. 5a and 5b illustrate an example primary actuation system.

FIG. 5c illustrates an example secondary actuation system.

DESCRIPTION

Preferred embodiments of the present disclosure may be describedhereinbelow with reference to the accompanying drawings. In thefollowing description, well-known functions or constructions are notdescribed in detail because they may obscure the disclosure inunnecessary detail. For this disclosure, the following terms anddefinitions shall apply.

As utilized herein the terms “circuits” and “circuitry” refer tophysical electronic components (i.e., hardware) and any software and/orfirmware (“code”) which may configure the hardware, be executed by thehardware, and or otherwise be associated with the hardware. As usedherein, for example, a particular processor and memory may comprise afirst “circuit” when executing a first set of one or more lines of codeand may comprise a second “circuit” when executing a second set of oneor more lines of code.

As utilized herein, “and/or” means any one or more of the items in thelist joined by “and/or”. As an example, “x and/or y” means any elementof the three-element set {(x), (y), (x, y)}. In other words, “x and/ory” means “one or both of x and y”. As another example, “x, y, and/or z”means any element of the seven-element set {(x), (y), (z), (x, y), (x,z), (y, z), (x, y, z)}. In other words, “x, y and/or z” means “one ormore of x, y and z”. As utilized herein, the term “exemplary” meansserving as a non-limiting example, instance, or illustration. Asutilized herein, the terms “e.g.,” and “for example” set off lists ofone or more non-limiting examples, instances, or illustrations.

As used herein, the words “about” and “approximately,” when used tomodify or describe a value (or range of values), mean reasonably closeto that value or range of values. Thus, the embodiments described hereinare not limited to only the recited values and ranges of values, butrather should include reasonably workable deviations. As utilizedherein, circuitry or a device is “operable” to perform a functionwhenever the circuitry or device comprises the necessary hardware andcode (if any is necessary) to perform the function, regardless ofwhether performance of the function is disabled, or not enabled (e.g.,by a user-configurable setting, factory trim, etc.).

As used herein, the terms “aerial vehicle” and “aircraft” refer to amachine capable of flight, including, but not limited to, bothtraditional runway and vertical takeoff and landing (“VTOL”) aircraft.VTOL aircraft may include fixed-wing aircraft (e.g., Harrier jets),rotorcraft (e.g., helicopters), and/or tilt-rotor/tilt-wing aircraft.

As used herein, the terms “communicate” and “communicating” refer to (1)transmitting, or otherwise conveying, data from a source to adestination, and/or (2) delivering data to a communications medium,system, channel, network, device, wire, cable, fiber, circuit, and/orlink to be conveyed to a destination. The term “database” as used hereinmeans an organized body of related data, regardless of the manner inwhich the data or the organized body thereof is represented. Forexample, the organized body of related data may be in the form of one ormore of a table, a map, a grid, a packet, a datagram, a frame, a file,an e-mail, a message, a document, a report, a list, or data presented inany other form.

Disclosed herein is a system configured to, inter alia, function as apilot's assistant (or co-pilot) or flight engineer. Such an aircrewautomation system may be configured to operate an aircraft from takeoffto landing, automatically executing the necessary flight and flight planactivities, checklists, and procedures at the correct phases of flight,while detecting contingencies and responding to them. At the same time,the pilot (e.g., a human pilot or operator) may be continuously informedthrough an intuitive human-machine interface operatively coupled withthe aircrew automation system. That is, the aircrew automation systemmay provide real-time information and/or feedback to the pilot. Forexample, the aircrew automation system may indicate a state of theaircraft relative to the procedure being accomplished. The aircrewautomation system may be configured to take back control of the aircraftthrough robotic actuators, if desired.

In doing so, the pilot is enabled to perform tasks best suited forhumans, such as high-level decision-making and flight plan planning.Tasks best suited for automation, however, may be handled by the aircrewautomation system, including, for example, the manipulation of controls,executing checklists, and monitoring aircraft engine and performancedata. Additionally, the aircrew automation system may have thecapability to access external information that is either currentlyunavailable to pilots or that which only comes with experience, such ascommon system failures for a particular aircraft or how air trafficcontrol commonly routes traffic at a particular airport. The aircrewautomation system may be configured to operate as an assistant or as theprimary pilot, thereby, if so configured, entirely obviating the needfor a human operator. Alternatively, the aircrew automation system mayserve to provide a pilot with continuous aircraft state monitoring andinformation augmentation, without actually taking control of theaircraft. For example, the aircrew automation system may serve as a“second set of eyes” for the pilot, monitoring checklists,instrumentation, engine state, airspace, flight regime, etc.

Unlike existing robotic autopilots and pilot assist systems, which areinvasive, require considerable installation expertise, and areaircraft-specific, an aircrew automation system in accordance with anaspect of the present disclosure employs a system architecture andknowledge acquisition system that enables rapid non-invasiveinstallation, which facilitates widespread use and enables the aircrewautomation system to be quickly adapted for use in a variety ofaircraft. Further, the aircrew automation system's data collection andperception systems are not limited to GPS, accelerations, orientation,and heading, as is the case with existing robotic autopilots. Indeed,the aircrew automation system exceeds the capability of existing datacollection and perception systems to better capture aircraft performanceby employing both standalone sensors, instrument image data capture(e.g., temperature, altitude, radar, flap angles, etc.), and measuring,detecting, or otherwise receiving pilot inputs. Further, the aircrewautomation system's core platform and design of the primary andsecondary flight control actuation systems enables portability across avariety of aircraft. Thus, unlike existing robotic autopilots or pilotassist systems, the aircrew automation system may be temporarilyinstalled and readily transferred from aircraft to aircraft, withoutinvasive modification to the aircraft. The aircrew automation system,through its modular design, further reduces the likelihood of designinga single point solution that becomes obsolete as aircraft evolve.

The aircrew automation system's combination of subsystems provides apilot with high-fidelity knowledge of the aircraft's physical state, andnotifies that pilot of any deviations in expected state based on, forexample, predictive models. This state awareness may be translateddirectly into useful information for the pilot, such as alerts todeveloping emergency conditions, fuel state computation, notification oficing conditions, etc. For example, the aircrew automation system mayalso serve as a digital flight engineer, thereby advising the pilot bymonitoring checklists, instrumentation, engine state, airspace, flightregime, etc.

This ride-along aircrew automation system, which may be non-invasivelyinstalled in preexisting aircraft, perceives the state of the aircraftvisually and via other sensors, derives the aircraft state vector andother aircraft information, and communicates any deviations fromexpected aircraft state to the pilot or a control tower. While theaircrew automation system may be non-invasively installed (e.g., via aperception system), it may alternatively be invasive. For example, theaircrew automation system may electronically couple with the cockpitinstrument panel (e.g., via the reverse side of the instrument panel)via, for example, the aircraft state monitoring system. Alternatively,the aircrew automation system may be integral and permanently installedduring fabrication of the aircraft. In conjunction with an actuationsystem, the aircrew automation system may further take control of theaircraft and autonomously navigate the aircraft.

System Level Architecture.

To share the duties and workload related to the execution of flightactivities, the aircrew automation system 100 should be capable ofexecuting the actions a pilot would perform routinely over the durationof a flight, regardless of the aircraft make, model, or type. An examplesystem architecture for an aircrew automation system 100 in accordancewith one aspect is shown in FIGS. 1a through 1c . As illustrated in FIG.1a , the core platform 102 may operate as a central subsystem thatconnects the other subsystems via one or more interfaces. The subsystemsmay communicate with one another through software and/or hardwareinterfaces 156 using wired and/or wireless communication protocols andhardware. FIG. 1b illustrates an example flow of information (e.g.,data) between the various subsystems.

The aircrew automation system 100 may comprise a core platform 102operatively coupled with a plurality of subsystems, such as those listedbelow. Each of the plurality of subsystems of the aircrew automationsystem 100 may be modular, such that the entire aircrew automationsystem 100 can be substantially ported to another aircraft rapidly. Forexample, the various subsystems may be removably and communicativelycoupled to one another via the core platform 102 using one or moresoftware and/or hardware interfaces 156. In certain aspects, however,the aircrew automation system 100 may alternatively be integral with theaircraft's system, thereby directly employing all sensors and indicatorsin the airplane. For example, the aircrew automation system 100, orcomponents thereof, may be integrated into the aircraft during itsdesign and manufacturing.

The plurality of subsystems may include, for example, a perceptionsystem 106, an actuation system 108, a human-machine interface (“HMI”)system 104, and a flight control system 116, each of which may beoperatively coupled with the core platform 102. In certain aspects, theneed for a perception system 106 may be mitigated or obviated via use ofanother aircraft state monitoring system. For example, the aircrewautomation system 100 may be coupled (e.g., communicatively orelectronically) with the instrument panel, or be otherwise integratedwith the aircraft or its systems. As can be expected, however, suchintegration would likely require a degree of modification to theaircraft or its wiring. The aircrew automation system 100 and/or coreplatform 102 may also comprise, or be operatively coupled to, aknowledge acquisition system 114 and a communication system 122. Themodular configuration further enables the operator to remove/disableunnecessary systems or modules or to add/install additional systems ormodules. For example, when the aircrew automation system 100 isconfigured to only provide information to the pilot via the HMI system104 (i.e., without the ability to control the aircraft), the actuationsystem 108 may be removed or disabled to reduce weight, cost, and/orpower consumption. Accordingly, depending on the configuration, theaircrew automation system 100 may be configured with fewer or additionalmodules, components, or systems without departing from the spirit andscope of the disclosure.

In operation, the flight control system 116 derives the aircraft statebased on information data from another subsystem (e.g., perceptionsystem 106) and directs another subsystem (e.g., the actuation system108) to operate (e.g., dynamically) in a manner to maintain aircraftstability. For example, the flight control system 116 may receivevehicle mode commands and configuration data from the core platform 102,while sending to the core platform 102 status and command informationgenerated by the flight control system 116. For example, the coreplatform may be configured to communicate one or more commands to theflight control system 116 of the aircraft based at least in part on theflight situation data, which may be obtained from the aircraft statemonitoring system 112, the perception system 106, or a combinationthereof.

The flight control system 116 may include, or communicate with, existingflight control devices or systems, such as those used in fixed wingaircraft and rotary wing aircraft. The communication system 122 enablesthe aircrew automation system 100 to communicate with other devices(including remote or distant devices) via, for example, a network. Thecommunication system 122 may receive communication commands andconfiguration data from the core platform 102, while sending to the coreplatform 102 status and response information from the communicationsystem 122.

Core Platform 102.

FIG. 2 illustrates an architecture diagram of an example core platform102. To enable a vehicle-agnostic aircrew automation system 100, a coreplatform 102 may provide, or otherwise serve as, middleware that can bemade specific to a particular aircraft or configuration through aninitial transition and setup phase. In other words, the mission controlsystem 110 may provide an operating system 206 that provides services toa set of operational applications 202 and output signals to one or moreof a set of hardware interfaces 204 or HMI system 104, while collectingand logging the data necessary to enable those applications.

The core platform 102 serves as the primary autonomous agent anddecision-maker, which synthesizes inputs from the perception system 106and HMI system 104 with its acquired knowledge base to determine theoverall system state. The core platform 102 may process inputs from thevarious sensor suites and aggregate the resultant information into anunderstanding of current aircraft state. The resultant information maybe compared against an aircraft specific file that encompasses aircrewautomation system's 100 understanding of pilot intent, system health,and understanding of appropriate aircraft procedures as they relate tothe aircrew automation system's 100 state estimation. The resultantstate knowledge and associated recommendations can be passed to a humanpilot via the HMI system 104 or, in certain aspects, to the flightcontrol system 116 and/or actuation system 108 to enable autonomousoperation. The aircrew automation system 100 may further generate a logof a given flight for later analysis, which may be used to facilitatepilot training that can provide detailed training and operations flightdebriefs. The logs may be used in connection with, for example, flightoperational quality assurance analysis, maintenance analysis, etc.

As illustrated, the core platform 102 may comprise a mission controlsystem 110 and flight controllers 118, each of which are configured tocommunicate with one another and the other subsystems via one or moresoftware and/or hardware interfaces 156, which may be a combination ofhardware (e.g., permanent or removable connectors) and software. Thecore platform 102 can host various software processes that track theaircraft and procedure states, as well as any modules for trendanalytics (predictive warnings) and machine learning routines. Incertain aspects, the aircrew automation system 100 and/or core platform102 may employ a computer bus and specification (e.g., as an interface)that facilitates discovery of a hardware component of a subsystem withinthe aircrew automation system 100 without the need for physical deviceconfiguration or user intervention in resolving resource conflicts. Sucha configuration may be referred to as “plug and play.” Thus, a user mayreadily add or remove system or subsystems (e.g., as modules) to theaircrew automation system 100 via the core platform 102 withoutrequiring substantive modification or integration efforts.

The core platform 102 outputs may be used to provide messages to the HMIsystem 104. The messages may indicate, for example, checklist progress,contingencies to initiate, warnings to raise, etc. The core platform 102may also contain a flight data recorder, for instance to provideperformance review capability and to provide robustness againstin-flight reset. The hardware and various computers may also beruggedized and share a housing with other devices, such as theperception computer. As discussed below, the core platform 102 may beoperatively coupled with a global positioning system (“GPS”)/inertialnavigation system (“INS”) system 154 and power management system (e.g.,28 VDC power). The core platform 102 may also contain a flight datarecorder, for instance to provide performance review capability and toprovide robustness against in-flight reset.

The mission control system 110 generally comprises a mission manager132, a standard interface 130 (e.g., a STANAG interface), a stateawareness manager 158, and other operational components 120 (e.g.,hardware and software controllers and/or interfaces), each of which arecommunicatively coupled to one another via one or more data buses 124.The open architecture of the core platform 102 enables the incorporationof additional data received from systems via the data bus 124. Incertain aspects, the mission control system 110 may be coupled with oneor more cockpit instruments of the aircraft via the vehicle systemsinterface to collect flight situation data. In other aspects, themission control system 110 may collect flight situation data through anaircraft state interface via the aircraft state monitoring system 112,which may collect or generate flight situation data via a directconnection to the aircraft and/or the perception system 106.

As illustrated, the mission control system 110 may be operationallycoupled with the secondary actuation system 108 b (e.g., when autonomousoperation is desired), the perception system 106, and the HMI system104, including the human-machine interface 126 (e.g., software and/orhardware that conveys inputs from and displays information to thepilot), and ground station 128. The mission control system 110 maycommunicate with the flight controllers 118 via the mission manager 132.

The flight controllers 118 may include, for example, an autopilotmanager 134 and a vehicle manager 136. The vehicle manager 136 may begenerally responsible for navigation and determining the location andstate of the aircraft. The vehicle manager 136 may be coupled with astate estimation module 142, which determines the estimated state of theaircraft using information received from the perception system 106 via aperception module 138 and from the GPS/INS system 154 via a navigationmodule 140.

The autopilot manager 134 may be generally responsible for controllingthe aircraft's flight based on, for example, information received fromthe vehicle manager 136 and the mission control system 110. Theautopilot manager 134 controls, inter alia, the flight control system152, which may be new or preexisting (and comprises a flight controller150), as well as the aircrew automation actuation module 144 and theaircraft actuation module 146. The aircrew automation actuation module144 may control the primary actuation system 108 a, while the aircraftactuation module 146 may control the aircraft controls 148 (e.g.,various flight surfaces and actuators).

In certain aspects, the flight controller's 118 components may overlapwith certain components of the flight control system 116. For example,in certain aspects (e.g., where redundancy is not desired andnon-invasive integration is possible), the core platform 102 may exploitcertain of the existing aircraft software and/or hardware, therebyobviating the need for additional hardware, such as certain flightcontroller 118 components and/or a GPS/INS system 154.

Open Architecture.

The core platform 102 serves as the central subsystem, or interface, ofthe aircrew automation system 100, connecting and controlling theremaining subsystems (e.g., as individual applications) in an openarchitecture. The remaining subsystems include, for instance, the flightcontrol system 116 (including any flight plan capabilities), the HMIsystem 104, the actuation systems 108 (e.g., the primary and secondaryactuation systems to provide autonomous operation where desired), theperception system 106, knowledge acquisition system 114, and othersubsystems 236. Thus, control of the other aircrew automation system 100hardware may be provided via separate applications specific to aparticular piece of hardware, which enables rapid integration of newsystems or other external flight plan support technology.

The core platform's 102 architecture enables rapid portability andextensibility when transitioning to a new aircraft or incorporating anew flight plan feature/capability. Thus, an application may be used toenable the aircrew automation system 100 to acquire informationspecific, or otherwise needed, for that aircraft or to provide the newcapability. For example, transition and setup can be handled byindividual applications that operate within the core platform 102 orother subsystems, representing aircraft-specific functionalities as wellas a growing library of capabilities of aircrew automation system 100,which can be exchanged depending on flight plan, aircraft or crewrequirements. In certain aspects, the transition process may besupported by software applications external to the aircrew automationsystem 100 (such as a procedure editor).

Aircraft Data Structure 208.

The operating system 206 operates as the middleware, interconnecting theoperational applications 202, hardware interfaces 204, and othersubsystems, such as the knowledge acquisition system 114. The operatingsystem 206 may employ an aircraft data structure 208, which may includea knowledge database 210, a procedure database 212, and a state database214.

The aircraft data structure 208 facilitates a vehicle-agnostic aircrewautomation system 100 by enabling the core platform 102 to develop acomplete understanding of an aircraft's systems, their configuration,and the procedures necessary to maintain safe operation, and all otherknowledge and expertise a certified pilot of that aircraft would beexpected to have. The aircraft data structure 208 may be populated bythe knowledge acquisition system 114 (discussed below), which containsnecessary information about the aircraft currently being operated (e.g.,flight control model, operational procedures, aircraft systems, etc.),data received from internal state sensors, and other subsystems orsensors.

The aircraft data structure 208 can be populated and adjusted to aspecific aircraft during a knowledge acquisition phase (e.g., duringinitial setup) such that it contains all the information necessary tooperate the aircraft. For example, when transitioning to a new aircraft,the knowledge acquisition system 114 may perform predefined activitiesin order to determine the layout (e.g., of the controllers/read outs,such as the cockpit instruments), performance parameters, and othercharacteristics of the aircraft. The predefined activities may include,for example: (1) generation of an aircraft system model, which informsaircrew automation system 100 about which systems are onboard and howthey are configured, actuation limits, etc.; (2) procedure codification,which informs aircrew automation system 100 how to operate aircraft innormal and non-normal situations, further including the codification ofchecklists; (3) an aerodynamic model, which informs the aircrewautomation system 100 how to fly the aircraft and what performance toexpect for which aircraft configurations; and (4) information aboutmission operations.

The core platform 102 can combine this information with data from a setof internal state sensors, which also improve redundancy and systemrobustness, thereby allowing the aircrew automation system 100 togenerate a highly accurate estimate of the aircraft state and systemstatuses, and to identify deviation from expected behavior. Duringflight operations, the data structure is dynamically updated withreal-time data gathered by, inter alia, the aircrew automation system's100, perception system 106, the HMI system 104, as well as the aircrewautomation systems 100 internal state sensing. Once the aircraft datastructure 208 for a given aircraft is populated, the aircraft datastructure 208 can then be retained in an aircraft library and used forall other aircraft of the same make and model for which aircrewautomation system 100 is available. The aircraft data structure 208 maybe further refined as additional data is generated and/or collected bythe aircrew automation system 100.

Operational Applications 202.

The core platform 102 may provide the aircrew automation system 100 witha plurality of operational applications 202. Examples of suchoperational applications 202 might include, without limitation, normalflight operation application 216, an anomaly detection application 218,a contingency operation application 220, an intelligence, surveillance,and reconnaissance (“ISR”) application 222 (e.g., ISR orbits), a trendrecognition application 238, or other flight plan-specific activityapplications 224, such as an aerial refueling application 316.

The normal flight operation application 216 enables aircrew automationsystem 100 to fly a predetermined flight plan from takeoff to landing,assuming no contingencies. The normal flight operation application 216is specific to the continuous execution of normal flight activity, asneeded by a particular flight phase. The predetermined flight plan maybe modified in flight due to unexpected disturbances such as weather,air traffic control commands, air traffic, etc.

The anomaly detection application 218 employs machine learningtechniques to monitor aircraft state, cluster, and classify sensorinputs in order to detect the presence of non-normal situations, and toidentify whether a contingency has occurred. The anomaly detectionapplication 218 is configured to compare the sensed states against a setof thresholds defined in the operational documentation for the specificaircraft (e.g., maximum parameters, such as never exceed a predeterminedairspeed, engine temperature, etc.). The anomaly detection application218 may also compare the sensed states against additional informationavailable to aircrew automation system 100 and generate alerts or othermessages in response to meeting predetermined or dynamically determinedthresholds (e.g., warning thresholds, etc.).

In case of a contingency condition, a contingency operation application220 executes the necessary predetermined checklists, procedures, andactions specified by the contingency operation application 220 in orderto maintain safe operation of the aircraft or safely divert the flight.Notably, if a departure from expected performance is observed, the pilotcan be alerted to a non-normal condition, thereby mitigating or avoidingpotential mistakes. If an aircraft is susceptible to a particularoperational error (e.g., pilot induced oscillations), the aircrewautomation system 100 can identify and mitigate such events. If ananomaly is detected, the contingency operation application 220 informsand interacts with the pilot via the HMI system 104 and ultimatelyexecutes the necessary procedure(s) to respond to the anomaly. Finally,the ISR application 222 and other flight plan-specific activityapplications 224 may provide instructions, algorithms, or information tocarry out operations relevant to a mission.

The trend recognition application 238 provides trend analysis developedusing machine learning based on, for example, the knowledge acquisitionsystem 114. In certain aspects, the trend recognition application 238may supply data, or otherwise trigger, the anomaly detection application218. For example, if the trend recognition application 238 detects anundesirable trend, the trend may be flagged as an anomaly and reportedto the anomaly detection application 218.

Hardware Interfaces 204.

The various information pertaining to the operational applications 202are communicated between the primary actuation system 108 a, secondaryactuation system 108 b, perception system 106, aircraft state monitoringsystem 112, HMI system 104, and other subsystems 236 via, for example,the primary actuation interface 226, secondary actuation interface 228,aircraft state interface 230, HMI interface 232, and other interface234.

Human/Machine Interface (HMI) System 104.

The HMI system 104 provides a control and communication interface forthe pilot (e.g., a human pilot, whether on-board or remote). The HMIsystem 104 is configurable to operate as a flight plan manager thatenables the pilot to direct the aircrew automation system 100. The HMIsystem 104 can combine elements of glass cockpits, unmanned aerialvehicle (“UAV”) ground stations, and electronic flight bags (EFB) toenable effective, efficient and latency-tolerant communication betweenthe pilot and aircrew automation system 100. Generally speaking, an EFBis an electronic information management device that allows flight crewsto perform a variety of functions that were traditionally accomplishedby using paper references. The HMI system 104 may include ahuman-machine interface 126, which may be based on a touch screengraphical user interface (“GUI”) and/or speech-recognition systems. Thehuman-machine interface 126 may employ, for example, a tablet computer,a laptop computer, a smart phone, or combination thereof. Thehuman-machine interface 126 can be secured near the pilot (e.g., on theyoke—as checklists often are, or on a knee-strap) depending on pilotpreferences. The human-machine interface 126 may be removable coupled tothe cockpit or, in certain aspect, employ an integrated display withinthe cockpit (e.g., an existing display).

FIG. 3a illustrates an example human-machine interface 126 having asingle-screen touch interface and speech-recognition system. The HMIsystem 104 serves as a primary channel of communication between thepilot and the aircrew automation system 100, enabling the pilot tocommand tasks to and receive feedback or instructions from the aircrewautomation system 100, to change the allocation of tasks between pilotand aircrew automation system 100, and to select which operationalapplications 202 are currently enabled for the aircrew automation system100.

As illustrated in FIG. 1b , for example, the HMI system 104 may receivestatus information from a subsystem via the core platform 102, whilesending to the core platform 102 mode commands generated by the HMIsystem 104 or input by the pilot. The pilot may be remote (e.g., on theground or in another aircraft) or on-board (i.e., in the aircraft).Thus, in certain aspects, the HMI system 104 may be remotely facilitatedover a network via communication system 122.

Human-Machine Interface 126.

As illustrated in FIGS. 3a and 3b , the human-machine interface 126 mayemploy a tablet based GUI and a speech-recognition interface thatenables vocal communications. An objective of the human-machineinterface 126 is to enable the pilot to interact with the core platform102's knowledge base in manner akin to the way a pilot interacts with ahuman flight engineer or copilot.

The human-machine interface 126 can display the current state of aircrewautomation system 100 (its current settings and responsibilities) aswell as which operational applications 202 are currently installed,which operational applications are running and, if they are active,which actions the operational applications 202 are taking. Thehuman-machine interface 126's GUI display may also be night-visiongoggles compatible such that it is visible regardless of the pilot'seyewear. The speech-recognition system may be used to replicate the sametypes of verbal communications used by human flight crew when runningthrough checklists and communicating on the flight deck. In certainaspects, the speech recognition may be limited to the same standards ofcodified communications used by pilot teams to minimize the chances ofthe system failing to recognize commands or changing into inappropriatemodes of operations. The speech-recognition system may be configured tolearn/recognize the speech of a given pilot through a voice trainingprotocol. For example, the pilot may speak a predetermined script suchthat the speech-recognition system can become trained with the pilot'sdialect.

The human-machine interface 126 may provide the status and/or details ofvarious operations, including the entire aircrew automation system 100via the aircrew automation status application 302, the perception system106 via the perception status application 304, the autopilot via theautopilot status application 306 (where applicable), the GPS/INS system154 via the GPS status application 308, and any other application orsystem status information 310. The display of the human-machineinterface 126 may be customized by the pilot. For example, the pilot maywish to add, reorganize, or remove certain of the display icons and/oroperational applications 202, which may be accomplished through a selectand drag maneuver or through the aircrew automation settings application312. The human-machine interface 126 may further inform the pilotregarding the aircraft's operating status and to provide the pilot withinstructions or advice.

As illustrated, the human-machine interface 126 may provide a tool barwith various selectable tabs, such as a route tab 328, a procedures tab330, a calibration tab 332, and an applications tab 334. When the pilotselects the applications tab 334, for example, the human-machineinterface 126 may display the various operational applications 202installed on the aircrew automation system 100 (e.g., the core platform102), including, for example, a normal flight operation application 216,a contingency operation application 220, an aircrew automation settingsapplication 312, gauge application 314, and aerial refueling application316.

Selecting the aircrew automation settings application 312 enables thepilot to change, reallocate, or otherwise edit the settings of theaircrew automation system 100 and/or to install operational applications202. Selecting the gauge application 314 may cause the human-machineinterface 126 to display the various operational conditions of theaircraft, including, for example, position, direction, speed, altitude,pitch, yaw, etc. The various operational conditions of the aircraft,which may be gathered from the perception system 106 or another sensor,may be displayed as alphanumeric characters or as graphical dials (e.g.,in accordance with the pilot's preference settings). Finally, selectingthe aerial refueling application 316 icon may cause the aircrewautomation system 100 to perform a predetermined protocol forfacilitating or coordinating a mid-air refueling operation. For example,upon selecting the aerial refueling application 316, the aircrewautomation system may coordinate with another aircraft to facilitaterefueling and perform the necessary checklists for doing the same (e.g.,ensuring aircraft position, airspeed, fuel hatch opening, etc.).Additional mission applications may be included that enable performanceof mission operations by the aircrew automation system.

When the pilot selects the route tab 328, the human-machine interface126 may display an area map 326 with an icon 322 representing thecurrent location of the aircraft along a flight path relative to itsvarious waypoints 320. Selecting (e.g., tapping, clicking, etc.) theicon 322 may cause a dialog window 324 to display that provides thevarious operational conditions of the aircraft. The area map 326 may besaved, exported, rotated, or panned using a map control window 318. Thearea map 326 may be saved or exported (e.g., via communication system122) as a static image or a data set (or database). When the pilotselects the calibration tab 332, the human-machine interface 126 maydisplay the calibration of the aircraft, whereby the pilot may befurther enabled to revise the same.

The HMI system 104 may provide an intuitive display and interface thatincludes checklist verification and health alerts from the core platform102 and predictions of aircraft state (e.g., fuel consumption andpredicted remaining range), as well as failure prognosis and deviationalerts (e.g., “Left engine EGT is 5 degrees above normal and rising”).Thus, when the pilot selects the procedures tab 330, as illustrated inFIG. 3b , the pilot may review and monitor checklist items, as well asreview any health alerts. Indeed, a function of the HMI system 104 is tofacilitate checklist monitoring and/or execution, marking items ascomplete when the when the perception system 106 perceives theircompletion and providing warnings to the pilot when items are notcompleted, as based on information previously imported from, forexample, a Pilot's Operating Handbook (POH). The aircrew automationsystem 100 also monitors system health, comparing the current systemstate to that expected based on the POH and other knowledge sources, andguides appropriate responses to contingencies. In certain aspects,either the pilot or the core platform 102 can acknowledge checklistactions as they are performed and the HMI system 104 automaticallyproceeds to the correct checklist as appropriate. The HMI system 104 maygive visual and auditory alerts to direct the pilot's attention tounattended checklist items, instruments that are displayingout-of-normal range values, or predicted events as the aircraft proceedsthrough the flight plan, which can be entered as a series of waypoints(for instance). For example, as illustrated, a list of tasks may beprovided alongside indicators that indicate whether the task has beencompleted, is being completed, or needs to be completed (e.g., a “checkmark” icon to include complete, an “in progress” icon, and a “to becompleted” icon). Similarly, a list of health hazards may be provide,along with one or corresponding icons to indicated one or moreoperational conditions that are out of range. For example, a low fuelindicator may be provided alongside a low fuel icon if fuel is low.

Task Allocation.

The HMI system 104 can enable the pilot to limit the activities executedby the aircrew automation system 100, if any. The HMI system 104 maydefine the allocation of tasks between the pilot and aircrew automationsystem 100, their responsibilities, and the communication of informationbetween the two, thereby functioning as a collaborative teammate of thepilot. Thus, the aircrew automation system 100 may operate, depending onconfiguration, in a purely advisory role (i.e., without any control overthe aircraft), a fully autonomous role (i.e., controlling the flightcontrol without pilot intervention), or an advisory role with theability to control flight controllers. The HMI system 104 may be furtherdesigned to enable a pilot to go through a transitional phase, where thepilot specifies the aspects of flight operation for which the aircrewautomation system 100 is responsible. For example, the HMI system 104may display a list of tasks where the pilot may select whether theaircrew automation system 100 or the pilot is responsible for a giventask on the list. The list of tasks may be provided to the HMI system104 from a procedure editor, which is described below. Once the aircraftdata structure 208 has been populated and refined such that the pilotbetter trusts the aircrew automation system 100, the pilot may allowaircrew automation system 100 to perform additional actions,transitioning the pilot from a primary mode to a supervisory mode (i.e.,a fully autonomous role). In this supervisory mode, pilot interactionsmay be at a high, goal-based level, with the HMI system 104 supportingthose tasks as well as allowing the operator insight at other levels fortroubleshooting. As noted above, in certain aspects, all tasks may beperformed by the pilot, leaving the aircrew automation system 100 toserve an advisory role.

Mode Awareness.

A risk when employing any automation system is the potential for modeconfusion on the part of the pilot (e.g., where the pilot neglects atask believing that the automation system will handle the task). The HMIsystem 104 avoids such mode confusion by first generating the correctfunction and the above-described task allocation between the aircrewautomation system 100 and the pilot. Indeed, the HMI system 104 allowsthe pilot to directly command and configure aircrew automation system100 via the human-machine interface 126 and displays the informationnecessary for the pilot to understand what actions the aircrewautomation system 100 is taking to ensure mode awareness. In otherwords, mode awareness generally refers to a state where the mode of thesystem matches the operational mode expected by the operator. Thehuman-machine interface 126 may display the information necessary toensure that the pilot is always aware of the mode in which aircrewautomation system 100 is operating. Additionally, the HMI system 104serves as the human interface for individual mission applications (e.g.,operational applications 202).

Aircraft State Monitoring System 112.

The aircraft state monitoring system 112 collects, determines, orotherwise perceives the real-time aircraft state. As noted above, theaircraft state monitoring system 112 may perceive the real-time aircraftstate through, inter alia, a direct connection (e.g., integral with orotherwise hardwired to the aircraft) to the aircraft, or via perceptionsystem 106. When a perception system 106 is used, the aircraft statemonitoring system 112 may include a dedicated controller (e.g.,processor) or share the perception controller 402 of the perceptionsystem 106. The perception system 106, for example, may employ acombination of a vision system, an acoustic system, and identificationalgorithms to read or comprehend flight situation information displayedby cockpit instruments. Example cockpit instruments include, forexample, an altimeter, an airspeed indicator, a vertical speedindicator, one or more compass systems (e.g., a magnetic compass), oneor more gyroscopic systems (e.g., attitude indicator, heading indicator,turn indicator), one or more flight director systems, one or morenavigational systems (e.g., very-high frequency omnidirectional range(VOR), non-directional radio beacon (NDB)), etc. The perception system106 may include a processor and one or more optical sensors (e.g., threeor more lightweight machine vision cameras) trained on the instrumentpanel to maximize pixel density, glare robustness, and redundancy. Theone or more optical sensors may wiredly connect to the perceptioncomputer via, for example, Ethernet. The one or more optical sensorsshould be installed with a line of sight with the instrument panel, butso as to be not obstructive to the pilot.

The flight situation data perceived by the perception system 106 may beencoded and provided to the core platform 102 in real-time. The openarchitecture of the core platform 102 enables the incorporation ofadditional data received via a data bus 124 to augment the flightsituation data generated by the perception system 106. As illustrated inFIG. 1b , for example, the aircraft state monitoring system 112 and/orthe perception system 106 may receive commands and configuration datafrom the core platform 102, while sending to the core platform 102status and flight situation information (e.g., flight situation data)gathered by the perception system 106 or otherwise collected by theaircraft state monitoring system 112.

FIG. 4 illustrates an example perception system 106 operatively coupledwith, inter alia, the core platform 102 (which is coupled to othersubsystems, such as flight control system 116), the GPS/INS system 154,and any other input systems 412. The perception system 106 visuallyand/or acoustically monitors, inter alia, the cockpit instruments togenerate flight situation data that can be used to derive the aircraftstate from cockpit layouts, which may range from basic analog aircraftinstruments to highly integrated, glass cockpit avionics suites. Inaddition to deriving physical state such as airspeed and altitude, theperception system 106 may also monitor instruments that are specific toaircraft systems such as fuel gauges and radios and provide secondaryfeedback about the status and positioning of the actuation system 108.

As illustrated, the perception system 106 may comprise a perceptioncontroller 402 that is operatively coupled with a database 404 and aplurality of sensors, such as cameras 410 (used for the vision system),microphone 408 (used for the acoustic system), and/or other sensors 406(e.g., temperature sensors, positional sensors, inertial sensors, etc.).The perception controller 402 may be, for example, a processorconfigured to feed flight situation data to (or otherwise instruct) thecore platform 102 based upon information received and manipulatedinformation received from the plurality of sensors, the database 404,and external components, such as the GPS/INS system 154 and other inputsystems 412.

Vision system. The perception system 106 may employ a monocular orstereovision system, possibly including motion capture markers, tocontinuously monitor the state of the aircraft by reading what isdisplayed on the cockpit instruments. In certain aspects, by comparinginformation about a scene from two vantage points, 3D information can beextracted by examining the relative positions of objects in the twopanels. The vision system may be used to accurately monitor instruments(e.g., glass gauges, physical steam gauges, etc.) and switches, as wellas their positions in a variety of lighting conditions and cockpitlayouts and sizes. Using a stereovision system and/or markers alsoprovides sensing to prevent collisions between any robotic componentsand the pilot.

The vision system may employ a suite of high-definition, stereo camerasand/or a LIDAR laser scanner. The system may be capable of recognizingdata from all flight instruments and derive the state of switches knobsand gauges that display the state of aircraft specific systems (e.g.,remaining fuel). It may also be capable of recognizing the state of thepanel with enough resolution to detect minor changes that result frompilot actions. Machine vision algorithms on the perception system 106computer ‘read’ the instruments (gauges, lights, wind correction anglepanel, individual elements of the primary flight display ormulti-function display in a glass cockpit) and mechanical items such asthrottle levers, trim settings, switches, and breakers to provide areal-time cockpit state update to the core platform 102.

The perception system 106 may be capable of deriving the aircraft statefrom cockpit layouts ranging from basic analog aircraft instruments tohighly integrated, “glass cockpit” avionics suites. Through the visionsystem, the requirement for a data feed from the aircraft is obviated,which permits/increases portability across aircraft. However, whenpossible, the aircrew automation system 100 may also be coupled to anaircraft's data feed (e.g., through a data port). Further, using theapplication approach described for the core platform 102, differentcockpit layouts can be addressed and understood using differentunderlying operational applications 202. For example, the aircrewautomation system 100 may employ the gauge application 314 to derive thevalues displayed on the instruments, whether graphical dial (e.g.,analog “steam” gauges or digital representations thereof) or a glasscockpit. This approach would also enable the aircrew automation system100 to run operational applications that monitor, inter alia, weatherradars, traffic displays, and terrain maps displayed in the cockpit.

In order to make aircrew automation system 100 portable, the process ofrapidly learning a new cockpit layout and codifying subtle differencesin location and scaling or unit of instruments is addressed by theperception system 106 design. For example, during the initial knowledgeacquisition phase, the location and scale of instruments and switchescan be encoded and verified for a particular aircraft, reducing thereal-time task to the extraction of the position of the graphical dial(round dial) or number (glass cockpit), whether graphical dial gauges,CRT display, LCD, etc. The piece-wise planar structure of cockpitinstrumentation enables the perception system 106 to construe the images(e.g., using Homography methods) and register it against the pre-mappeddata generated during the initial knowledge acquisition phase.Accordingly, live imagery can be registered and compared against thepreviously annotated model, thereby greatly simplifying interpretationof the data.

Actuation System 108.

When desired, an actuation system 108 executes the actions commanded viathe core platform 102 to guide the flight and overall operation of theaircraft. The aircrew automation system's 100 actuation system 108executes the actions commanded by the core platform 102 to guide theflight and overall operation of the aircraft without interfering withthe activities performed by the pilot. As illustrated in FIG. 1b , forexample, the actuation system 108 may receive actuation commands andconfiguration data from the core platform 102, while sending to the coreplatform 102 status and response information generated by the actuationsystem 108.

Manned aircraft cockpits are designed for the human reach envelope and,therefore, all cockpit controls are reachable by a comparably sizedrobotic/mechanical manipulator. A manipulator capable of actuating everysingle switch, knob, lever and button on every single possible cockpitin high-G and vibration environments with the rapid execution requiredfor emergency operation, however, would be expensive, heavy, and moreinvasive than what is desired for the aircrew automation system 100.

To more effectively achieve portability across aircraft, the aircrewautomation system 100 may separate the actuation of primary flightcontrols (stick/yoke, stick, side-stick or collective, rudder pedals,brakes, and throttles) from the actuation of secondary flight controls(e.g., switches, knobs, rockers, fuses, etc.). This approach reduces thelikelihood of designing a single point solution that becomes obsolete asaircraft evolve. Thus, the aircrew automation system 100 may employ aprimary actuation system 108 a and a secondary actuation system 108 b tophysically control the actuators in the cockpit. More specifically, theprimary actuation system 108 a may actuate the primary flight controls,while the secondary actuation system 108 b may actuate the secondaryflight controls, without obscuring the use of those controls by thepilot. The primary actuation system 108 a and the secondary actuationsystem 108 b are configured to collectively actuate all standardcontrols present on today's flight decks during flight operations.

As discussed below, the primary actuation system 108 a focuses onactuating the primary flight controls (stick/yoke, stick, side-stick orcollective, rudder pedals, brakes, and throttles), while the secondaryactuation system 108 b focuses on actuating the controls that are not aseasily accessed by the primary actuation system 108 a, such as secondaryflight controls (e.g., switches, knobs, rockers, fuses, etc.).

Primary Actuation System 108 a.

The primary actuation system 108 a focuses on the set of controlsnecessary to safely operate the aircraft. As shown in FIGS. 5a and 5b ,primary actuation system 108 a include a frame 516 having anarticulating arm 502 (e.g., a robotic appendage or “arm”) and stick/yokeactuator 510 that actuates the primary flight controls (yoke, stick,side-stick or collective, rudder pedals, brakes, and throttles) andother, easy to reach controls. The actuators may be one or more oflinear (straight line), rotary (circular), or oscillatory actuators,which may be driven through one or more of electrical, pneumatic, and/orhydraulic techniques.

The frame 516 may be sized and shaped to fit within the seat of astandard aircraft. To that end, the frame's 516 footprint should beabout the same size as, or smaller than, an average human's “seated”footprint. The actuation system 108 may be fabricated using lightweightmetals, metal alloys, and/or composite materials.

Stick/Yoke Actuator 510.

The stick/yoke actuator 510 may couple to and engage the aircraft'sexisting stick/yoke 514 using a stick/yoke gripper 512. The stick/yokegripper 512 may be sized and shaped such that it is universal and canengage various forms of stick/yokes and/or control wheels. Thestick/yoke actuator 510 may be configured to move the stick/yoke 514forward, backward, left, right, and positions therebetween. Thestick/yoke gripper 512 may further comprise one or more actuators foractuating buttons and/or switches positioned on the stick/yoke 514.

Articulating Arm 502.

The actuator-controlled articulating arm 502 may be sized, shaped, andconfigured to occupy the space typically occupied by a co-pilot's arms,thereby ensuring portability across aircraft. To enable movement inmultiple degrees of freedom (“DOF”) movement, the articulating arm 502may comprise a plurality of arm segments (whether linear, curved, orangled) joined using a plurality of hinged or pivotal joints 506. Thearticulating arm 502 may comprise a gripper 504 at its distal end. Thegripper 504 may be coupled to the articulating arm 502 via amultiple-DOF connection. The base of the articulating arm 502 may berotatable and slideably coupled to the frame 516 via a movable base 508.For example, the articulating arm 502 may be coupled with an upper base508 a, which is slideably coupled with a lower base 508 b, which may besecured to the frame 516. The articulating arm 502 and/or frame 516 canbe grounded to the aircraft electrically via a grounding lead 518 (e.g.,cable). The upper base 508 a may slide relative to the lower base 508 busing, for example, a combination of rails and ball bearings. In certainaspects, the upper base 508 a may slide relative to the lower base 508 balong both the X- and Y-axis.

The articulating arm 502 can be equipped with an encoder (e.g., twin18-bit encoders) for each of its degrees of freedom to ensure exactpositioning of the articulating arm 502. Internal clutches may beprovided at each hinged or pivotal joint 506 such that the articulatingarm 502 can be overpowered by the pilot if so desired, without damagingthe articulating arm 502. In such a case, the aircrew automation system100 may determine the position or location of the articulating arm 502using the encoders.

The gripper 504 may be configured to couple, or otherwise engage, forexample, throttle levers, etc. The gripper 504 may also provide forceand pressure detection so as to allow the aircrew automation system 100to estimate how a flight controls actuator is grasped and to adjust themotion to properly throw it. Once the motion is executed, the samefeedback may be used to determine if the desired switch configurationhas been achieved. In certain aspects, the articulating arm 502 may befitted with an electronic device (e.g., a homing device) that enables itto find and hit a target.

Secondary Actuation System 108 b.

Unlike the primary flight controls, which are generally located in thesame vicinity across aircraft makes and types, the location of thesecondary flight controls (e.g., avionics, switches, knobs, rockers,toggles, covered switches, fuses, etc.) is not as consistent orspatially contained from aircraft to aircraft.

The secondary actuation system 108 b focuses on actuating the controlsthat are not as easily accessed by the primary actuation system 108 a.For example, some switches may even be on an overhead panel directlyabove the captain's head, making it potentially difficult to manipulatethem with a robotic arm (especially in turbulent flight conditions).Accordingly, some actuators may be allocated to the above-describedprimary actuation system 108 a, while others may be allocated to aself-contained, secondary actuation system 108 b.

The secondary actuation system 108 b may be provided in the form of anadaptable XY-plotter or gantry system mounted directly to the panel ofinterest and calibrated to the specific panel it is operating. Thesecondary actuation system 108 b is preferably universal and resizable.An example XY-plotter is illustrated in FIG. 5c . The XY-plotter maycomprise a square frame that serves as the rails 520 of the plotter, arotatable multi-tool 528 with multiple interfaces (e.g., switch actuator532 and knob actuator 530) capable of manipulating the controls ofinterest, and a control system that moves this multi-tool 528 via amulti-tool carriage 526 within the frame along a Y-axis set of rails 522and an X-axis set of rails 524.

When in use, the plotter moves the multi-tool 528 to the location,selects the correct manipulator interface, and manipulates the secondaryflight control of interest. For example, the multi-tool 528 that canflip binary switches and/or covered switches using a switch actuator 532and can twist knobs using a knob actuator 530. The switch actuator 532and/or knob actuator 530 may be coupled to the multi-tool 528 via anarticulating or rotating member, such as the rotatable switch arm 534.

When not in use, the multi-tool carriage 526 may return to a homeposition (e.g., automatically navigate to a far corner) to preventobstruction of the panel. The multi-tool 528 and/or multi-tool carriage526 would be equipped with sensors (e.g., proximity sensors) such thatit can move out of the way when it detects the pilot's hands. During theinitial set-up of the plotter on a new aircraft, the location, type, andposition of the secondary flight control panel may be encoded. Once aparticular secondary flight control panel is encoded, the configurationcan be saved to the aircraft data structure 208 and loaded when aircrewautomation system 100 is installed in the same aircraft, or the sametype of aircraft. In certain aspects, additional actuators may beprovided to actuate controllers that are positioned in, for example, thefoot well of the cockpit, such as foot pedals (e.g., brake and/or rudderpedals).

Knowledge Acquisition System 114.

The knowledge acquisition system 114 gathers and/or generates aknowledge base necessary to enable the aircrew automation system 100 todetermine aircraft specific information. This includes knowledge ofaircraft performance characteristics, limitations, checklists, andprocedures (including emergency procedures), and criteria that definecontingencies in the aircraft. The data may be derived from acombination of encoded data (e.g., from manuals, pilot briefings, pilotoperating handbook) and data acquired in flight (e.g., via sensors),which supports off-line machine-learning and trend analysis. The date tobe encoded may be loaded in .xml format that describes the contents ofprocedures and the flow of tasks both within and between procedures.

As illustrated in FIG. 1b , for example, the knowledge acquisitionsystem 114 may receive operational commands from the core platform 102,while sending to the core platform 102 configuration data and status andresponse information generated by the knowledge acquisition system 114.

The operation of the knowledge acquisition system 114 may be generallydivided into three processes, including, for example, aircraft systemmodeling, procedure codification, and aerodynamic modeling. The aircraftsystem modeling process provides the aircrew automation system 100 withinformation about the available onboard systems and how the onboardsystems are configured, actuation limits, etc. The procedurecodification process provides the aircrew automation system 100 withinformation about aircraft operation in normal and non-normalsituations. Procedure codification may include, for example, thecodification of checklists. Finally, aerodynamic modeling processprovides the aircrew automation system 100 with information about flyingthe aircraft and what performance to expect for a given aircraft typeand configuration.

During the knowledge acquisition phase, the conditions under which asituation is considered an anomaly or contingency must also beestablished. These conditions will frequently be discrete, such as anengine over-speed or the exceedance of an airspeed limit. Using machinelearning, the aircrew automation system 100 can fine-tune itsaerodynamic and control models by observing a series of in-flightmaneuvers flown by the pilot. This information includes flight dynamicsdata, operational limitations, procedures, aircraft systems, and layoutsas well as other related data. In addition to written information, theaircrew automation system 100 may also codify information based on pastevents and experience of more experienced pilots. Machine learningenables the knowledge acquisition process to be performed efficientlyand quickly.

Using aircrew automation system's 100 perception system 106 andactuation system 108, the instruments and controls in a plane cockpit ora realistic simulator are monitored as a pilot goes through the motionsof a typical flight profile. Observing the pilot's actions allows theaircrew automation system 100 to learn directly from the pilot andimitate the smooth, expert control for a given operation. This processbenefits from the fact that flight operations are highly structured inwhat is to be done in a given situation—machine learning then enablesthe codification of how something is to be executed.

The population of aircraft data structure 208 may be accomplished usingthe Extensible Markup Language (“XML”). More specifically, a XML datastructure may be employed that comprises a set of fields and data treesthat, when populated, allow the core platform 102 to configure andoperate an aircraft. In certain aspects, the aircrew automation system100 may employ natural language interpretation of flight documentsand/or a software tool that enables a human to enter the dataefficiently and accurately.

In certain aspects, a set of airplane agnostic features may be generatedand coded. For example, procedures like landing gear retraction, engineout procedures on multi-engine aircraft, and stall recovery are similaracross many types of aircraft and will need only minimal modificationfor a particular airframe. Moreover, basic airframe limitations (such asnever exceed speeds) need only be entered as specific numbers and can beentered from flight manuals in a nominal period of time.

Procedure Editor.

The aircraft specific information may be gathered during a transitionperiod using, for instance, written documentation (e.g., pilot operatinghandbook, maintenance manual, etc.) as well as through direct monitoringof aircraft operations. The output of this knowledge acquisition processis the aircraft data structure 208, which is described above with regardto the core platform 102. Contained in this aircraft data structure 208may be operational procedures, available systems and their designs,cockpit layout, and all other information necessary for safe operationof the aircraft. In certain aspects, an aircrew automation softwaredevelopment kit may allow a software/flight controls engineer tospecify, code, and unit-test one aircraft subsystem (e.g., electrical orhydraulic) per day. The aircrew automation software development kit canprovide tools for turning the flight manual's procedures into statemachines compatible with Matlab State Flow™ and Simulink™ programs,which can then auto-code the procedures in C for inclusion in the coreplatform 102. The aircrew automation software development kit may alsogenerate test code for the unit-level as well as interfaces for testingto the core platform 102. For example, the procedure editor may providea list of tasks where the pilot may select whether the aircrewautomation system 100 or the pilot is responsible for a given task onthe list.

Knowledge Acquisition of Flight Control.

A first step in knowledge acquisition of flight control uses the AthenaVortex Lattice (“AVL”) method to generate the mathematical model in theform of non-dimensional stability derivatives that is used and refinedduring the flight with the pilot. Once the primary flight controlmechanisms are calibrated, the system ID trainer application may be usedto perform a sequence of flight maneuvers designed to identify specificstability derivatives. The data is automatically processed into updatedstability derivatives for use in the controller. The controller mayemploy an auto-tuner. The same updated stability derivatives are used ina 6-DOF simulation as a validation step that the controllers performadequately prior to flight. An additional benefit of performingknowledge acquisition of flight control is that it enables therefinement and incorporation of a great deal of formal proceduralknowledge. Although the procedures lay out the individual steps, finedetail on how such steps are to be executed may be missing (e.g., howlong to wait between steps or how sharply to increase throttle).

Reverse Engineer of Aircraft Flight Performance Characteristics.

Aircraft performance characteristics that can be measured throughon-board data-acquisition units are generally considered proprietary bythe aircraft and avionic manufacturers. This information can be utilizedfor flight simulations, aircraft health monitoring, aircraftdevelopment, and much more. Currently, third parties wanting to utilizethe on-board data acquisition are restricted by its proprietary nature.This restriction has only been partially been overcome through the useof stand-alone aircraft sensor suites. These commercially availablesensor suites only measure a fraction of the data available throughcockpit instrumentation and pilot inputs. However, because the aircrewautomation system 100 utilizes a variety of sensors to determine theaircraft flight performance characteristics, it effectively reverseengineers the air vehicle performance characteristics. The aircrewautomation system 100 collects aircraft information through acombination of stand-alone sensors, data capture through images ofcockpit instrument, and input controls.

Example

Aspects of the present disclosure may be illustrated through thefollowing example flight plan, which illustrates how aircrew automationsystem 100 may interact with the pilot, execute a flight plan, executeflight operational tasks, and respond to contingencies during systemengagement and takeoff, flight plan engagement, and anomaly detection &handling. The present teachings, however, should not be limited to thoseused in this example.

System Engagement and Takeoff.

The pilot gets into the left seat of an aircraft, fastens the seat belt,positions the human-machine interface 126 comfortably at his side, andactivates the aircrew automation system 100 application. The applicationboots and runs through a series of power-on diagnostics and themechanical interfaces power up and calibrate. A message may be displayedupon the human-machine interface 126 confirming a successful test andqueries the pilot to confirm engagement of aircrew automation system100. The pilot selects the day's flight plan via the applications tab334. The aircrew automation system 100 may be used for checklistmonitoring. The pilot selects engine start, and aircrew automationsystem 100 may begin a sequence of engine start actions, asking forfinal confirmation before actually starting. Meanwhile, the pilot maycall the tower for clearance and receives a flight plan to the trainingarea.

When engine start is complete, the aircrew automation system 100 mayreport success to the pilot and report, for example, “ready to taxi,”(either audibly or via the human-machine interface 126). The pilot callsfor a taxi clearance and upon hearing it, the aircrew automation system100 transcribes the taxi clearance and displays it to the pilot forconfirmation. The pilot then hits the “taxi via clearance” button on theapplication and aircrew automation system 100 taxis to the assignedrunway while the pilot monitors for traffic. When at the runwaythreshold, the pilot verbally commands the aircrew automation system 100to perform a pre-takeoff check and the system completes all necessarychecks, prompting the pilot to manually double-check critical items,such as flight controls. For example, the aircrew automation system 100may monitor the human operator's execution of a checklist, and output“checklist complete” or identify a flight plan or error.

Upon receiving further clearance, the pilot then commands the aircrewautomation system 100 to guide the aircraft to line-up and wait, andthen ultimately takeoff. The aircrew automation system 100 pushes thethrottles forward via the primary actuation system 108 a, visuallychecks engine and cockpit indicators via the perception system 106,calls out speeds via the HMI system 104, and rotates at the speedappropriate to the current weight, balance, and density altitude. Thepilot keeps his hand on the stick/yoke 514 to confirm aircrew automationsystem's 100 inputs and retain his muscle memory. The aircrew automationsystem 100 confirms aircraft performance according to current conditionsand reports any deviation from expected climb rate. The pilot's workloadis reduced by the aircrew automation system 100 during climb, enablingmore heads-up time (i.e., eyes forward, not on the instruments) to lookfor traffic in the busy airspace. The aircrew automation system 100 mayalso provide experienced pilot advice for a given checklist, aircraft,or location. For example, in a particular airport, the aircrewautomation system 100 may instruct the human operator with airportspecific tips, such as “steep departure angle from this runway.”

Flight Plan Engagement.

At the top of climb, the aircrew automation system 100 levels off theaircraft and adjusts trim and power settings while heading to the firstwaypoint in the flight plan. During cruise, the aircrew automationsystem 100 continues to visually monitor all cockpit displays,constantly comparing engine and aircraft performance against expectedvalues and alerting the pilot to any deviations.

The aircraft arrives at the training area and begins the day's flightplan. During the flight plan, however, the aircraft enters a toweringcumulus cloud, where instrument meteorological conditions (“IMC”)conditions are at below freezing temperatures. The pilot requests andreceives clearance from the ground, via an internet relay chat (“IRC”)chat window on the human-machine interface 126, to climb to 24,000 feetto get above the weather. In certain aspects, the aircrew automationsystem 100 request clearance from the ground.

Anomaly Detection & Handling.

After a period of time, the aircrew automation system 100 may detectthat given the climb, the indicated airspeed is slowly deviating fromits modeled airspeed for these pitch and power settings, indicatinglower than expected values. This is an indication that the pitot heaterhas failed and the pitot tubes have iced up. The pilot has fewer than100 hours flying the aircraft and is not aware that pitot heaters inthis model are known to be unreliable. The pilot has not yet noticedthat the airspeed indicator is trending below nominal.

The aircrew automation system 100, however, recognizes that the airspeeddata is anomalous to the rest of the flight data and its internal flightdynamics model, and aurally warns the pilot “airspeed indicator fault.”While the pilot recognizes that the airspeed information is currentlyunreliable, he is unsure as to whether the aircraft is flying faster orslower than the indicator shows.

Drawing on a database of prior anomalies, the aircrew automation system100 presents a set of procedural options and highlights the minimum safealtitude for the area (e.g., 8,000 ft). The pilot chooses the mostconservative option, which results in wings level, pitch, and powerdescent to a lower altitude (e.g., 10,000 ft). The aircrew automationsystem 100 eases back on the power, pitches slightly down, and commencesthe descent. While descending through 15,000 feet the pitot tubes againcome online. Once stable at 10,000 feet, the aircrew automation system100 holds the aircraft straight and level while the pilot assesses thesituation prior to returning to the flight plan.

Upon competition of the day's flight plan, the aircrew automation system100 may execute an automatic landing procedure. For example, the aircrewautomation system 100 may navigate the aircraft to a predeterminedwaypoint, where the aircraft may commence its initial descent. Duringthe descent, the aircrew automation system 100 may monitor the flightconditions and locate the runway. Upon final approach, the aircrewautomation system 100 may slow the aircraft down and ultimately land theaircraft. If the aircrew automation system 100 determines that landingis not feasible (e.g., an obstruction or unacceptable flightconditions), the aircrew automation system 100 may commence a missedapproach routine or other contingency routine. For example, the aircrewautomation system 100 may retry landing at the same location or navigatethe aircraft to an alternative landing location. An example system forlanding an aircraft at an alternative landing location is disclosed bycommonly owned U.S. Patent Publication No. 2015/0323932, titled“Autonomous Cargo Delivery System.”

The aircrew automation system 100 and derivative technologies may beapplied across a wide range of aircraft and flight simulators. Thederived flight performance characteristics from an aircraft flight testcan be used to improve the fidelity of flight simulators used to trainpilots. Providing flight simulators access to actual aircraftperformance data has tremendous value for flight simulator operators.Another benefit of aircrew automation system 100 is its ability tosynthesize flight performance characteristics when aircraft are modifiedfor special flight plans such as the addition of sensors and antennasthat can affect aerodynamic performance and flight handling qualities(e.g., aircraft development). In addition, the data captured by theaircrew automation system 100 can be used for aircraft healthmonitoring, using prognostics to sense maintenance needs.

The aircrew automation system 100 furthers the safety and utility ofcommercial aviation operations while providing significant savings inhuman operating costs. For example, the aircrew automation system 100may be applied to long-haul air cargo carriers to increase safety andefficiency as well the cost-savings of this advanced pilot-assisttechnology. Further, the ultimate state machine, for example, may serveas a training tool for pilots in-flight, or as a safety system,providing a second set of eyes in what would traditionally be asingle-pilot aircraft. Portions of the human-machine interface 126streamline all piloted flight operations, even multi-crew operations.

The above-cited patents and patent publications are hereby incorporatedby reference in their entirety. Although various embodiments have beendescribed with reference to a particular arrangement of parts, features,and like, these are not intended to exhaust all possible arrangements orfeatures, and indeed many other embodiments, modifications, andvariations may be ascertainable to those of skill in the art. Thus, itis to be understood that the claimed invention may therefore bepracticed otherwise than as specifically described above.

1. A method of operating an aircraft using an aircrew automation systemhaving one or more processors and an optical perception system, themethod comprising: imaging, via the optical perception system, cockpitinstruments of the aircraft to yield, in real-time, image datarepresenting the cockpit instruments; generating, via the one or moreprocessors, real-time flight situation data based on the image data;generating, via the one or more processors, one or more flight commandsbased at least in part on the real-time flight situation data;communicating the one or more flight commands to a flight control systemof the aircraft to control at least one flight control of the aircraft;and displaying, via a display device operatively coupled with the one ormore processors, a status of the aircraft and of the aircrew automationsystem.
 2. The method of claim 1, further comprising the step ofdisplaying, via the display device, at least one of a checklist, ahealth alert, a predicted aircraft state, and a failure prognosis. 3.The method of claim 1, further comprising the step of receiving, via atouch screen of a human-machine interface, one or more pilot commandsfrom a human pilot to control the aircrew automation system.
 4. Themethod of claim 1, wherein the real-time flight situation data includesairspeed, altitude, and vertical speed of the aircraft.
 5. The method ofclaim 1, further comprising the step of determining, via the one or moreprocessors, aircraft-specific information for the aircraft during aknowledge acquisition phase, wherein the aircraft-specific informationincludes a layout of cockpit instruments for the aircraft.
 6. The methodof claim 5, further comprising the step of, during the knowledgeacquisition phase, receiving a flight plan from a flight control systemof the aircraft, wherein the one or more flight commands are generatedbased at least in part on the flight plan and the real-time flightsituation data.
 7. The method of claim 5, wherein the aircraft-specificinformation further comprises one or more thresholds defined for theaircraft.
 8. The method of claim 7, wherein the one or more thresholdsis a maximum airspeed or maximum engine temperature.
 9. The method ofclaim 1, further comprising the step of receiving flight data directlyfrom the cockpit instruments via a data link, wherein the real-timeflight situation data is based at least in part on the flight data. 10.An aircrew automation system for use in an aircraft, the aircrewautomation system comprising: one or more processors to operativelyconnect a plurality of systems or subsystems via one or more interfaces;a perception system operatively coupled with the one or more processorsto monitor, in real-time, one or more cockpit instruments of theaircraft visually to generate real-time flight situation data, whereinthe one or more processors are configured to generate one or more flightcommands based at least in part on the real-time flight situation data,and wherein at least one of the one or more processors is configured tocontrol at least one flight control of the aircraft using the one ormore flight commands; and a human-machine interface operatively coupledwith the one or more processors to provide an interface between a humanpilot and the aircrew automation system, wherein the human-machineinterface comprises a display device to display a status of the aircraftand of the aircrew automation system.
 11. (canceled)
 12. The aircrewautomation system of claim 10, wherein the human-machine interface isconfigured to display a plurality of tasks on a touch-screen display ina form of a task list, wherein each of the plurality of tasks is markedas either completed or not completed based at least in part on an inputfrom the human pilot via a touch-screen display.
 13. The aircrewautomation system of claim 10, wherein the one or more processors areconfigured to receive and aggregate the real-time flight situation datato determine a current aircraft state for the aircraft.
 14. The aircrewautomation system of claim 10, further comprising an actuation systemconfigured to mechanically engage the at least one flight control,wherein the actuation system is configured to manipulate the at leastone flight control in response to the one or more actuation commands.15. The aircrew automation system of claim 10, wherein the perceptionsystem comprises a vision system or an acoustic system, wherein theperception system is configured to read or comprehend flight situationinformation provided by the cockpit instruments using the vision systemor the acoustic system.
 16. The aircrew automation system of claim 10,wherein the human-machine interface comprises a microphone and employsspeech-recognition techniques to recognize a verbal command from thehuman pilot.
 17. The aircrew automation system of claim 16, wherein thehuman-machine interface is configured to generate a verbal reply inresponse to the verbal command from the human pilot.
 18. The aircrewautomation system of claim 10, further comprising a knowledgeacquisition system operatively coupled with the one or more processorsto determine aircraft-specific information for the aircraft during aknowledge acquisition phase.
 19. The aircrew automation system of claim10, wherein the one or more processors are configured to identify ananomaly in a behavior of a human pilot or an operation of the aircraftusing machine learning, wherein, upon detection of the anomaly, the oneor more processors executes a contingency operation to adjust control ofthe aircraft.
 20. The aircrew automation system of claim 10, wherein theone or more processors are configured to identify an anomaly as afunction of the real-time flight situation data and to generate an alertupon detection of the anomaly.